feat(catalog-etl): account for full runEtl() runs in scheduler skip guard#292
Open
GitAddRemote wants to merge 1 commit into
Open
feat(catalog-etl): account for full runEtl() runs in scheduler skip guard#292GitAddRemote wants to merge 1 commit into
GitAddRemote wants to merge 1 commit into
Conversation
- Extend getLastSuccessfulStepRun() WHERE clause to also match step_name IS NULL rows (full runEtl() runs) in addition to per-step runStep() rows; steps_failed = 0 and no error warning condition already ensures all steps succeeded in a full run - Add 3 tests: null result, step-name match, and query contains 'r.step_name IS NULL' Closes #228
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
getLastSuccessfulStepRun()previously filteredWHERE r.step_name = $1, which only matchedrunStep()invocations — fullrunEtl()rows havestep_name = NULLand were never considered(r.step_name = $1 OR r.step_name IS NULL), so a clean full-ETL completion within 12h correctly triggers the skip guard forterminals-syncandterminal-distances-syncsteps_failed = 0andNOT EXISTS (severity='error' warning)guards remain, ensuring partial or failed full runs don't suppress individual step rerunsNote: This PR targets
feature/ISSUE-227(notmain) since it stacks on top of the skip-guard safety wrapper introduced there.Test plan
nullwhen no completed run existscompleted_atfrom a step-name-matched runr.step_name IS NULL(verifies the query was actually updated)catalog-etl.service.spec.tstests passCloses #228